Search for: All records

This study explores the potential of the large language model GPT-4 as an automated tool for qualitative data analysis by educational researchers, exploring which techniques are most successful for different types of constructs. Specifically, we assess three different prompt engineering strategies — Zero-shot, Few-shot, and Few-shot with contextual information — as well as the use of embeddings. We do so in the context of qualitatively coding three distinct educational datasets: Algebra I semi-personalized tutoring session transcripts, student observations in a game-based learning environment, and debugging behaviours in an introductory programming course. We evaluated the performance of each approach based on its inter-rater agreement with human coders and explored how different methods vary in effectiveness depending on a construct’s degree of clarity, concreteness, objectivity, granularity, and specificity. Our findings suggest that while GPT-4 can code a broad range of constructs, no single method consistently outperforms the others, and the selection of a particular method should be tailored to the specific properties of the construct and context being analyzed. We also found that GPT-4 has the most difficulty with the same constructs than human coders find more difficult to reach inter-rater reliability on. 

Research on epistemic emotions has often focused on how students transition between affective states (e.g., affect dynamics). More recently, studies have examined the properties of cases where a student remains in the same affective state over time, finding that the duration of a student's affective state is important for multiple learning outcomes. However, the likelihood of remaining in a given affective state has not been widely studied across different methods or systems. Additionally, the role of motivational factors in the persistence or decay of affective states remains underexplored. This study builds on two prior investigations into the exponential decay of epistemic emotions, expanding the analysis of affective chronometry by incorporating two detection methods based on student self-reports and trained observer labels in a game-based learning environment. We also examine the relationship between motivational measures and affective decay. Our findings indicate that boredom exhibits the slowest decay across both detection methods, while confusion is the least persistent. Furthermore, we found that higher situational interest and self-efficacy are associated with greater persistence in engaged concentration, as identified by both detection methods. This work provides novel insights into how motivational factors shape affective chronometry, contributing to a deeper understanding of the temporal dynamics of epistemic emotions. 

Abstract This study investigates student learning and interest within the context of a single-player, open-world game designed for microbiology inquiry. The game immerses players in the role of investigative scientists tasked with diagnosing a mysterious illness on a remote island. Ordered Network Analysis (ONA) was combined with clustering techniques to analyze in-game actions (i.e., interactions with non-playable characters, exploration, and utilization of in-game educational tools) allowing us to construct student archetypes based on the behavioral patterns of 122 middle schoolers. The analysis identified four distinct clusters of students with varying engagement patterns—two showing apparent patterns of engagement and two showing apparent patterns of disengagement. The study contributes insights into tailoring educational game designs to address disengaged or ineffective behaviors, enhancing the efficacy of game-based learning experiences. 

The artificial intelligence in education (AIED) community has produced technologies that are widely used to support learning, teaching, assessment, and administration. This work has successfully enhanced test scores, course grades, skill acquisition, comprehension, engagement, and related outcomes. However, the prevailing approach to adaptive and personalized learning has two main steps. First, the process involves detecting the areas of knowledge and competencies where students are deficient. This process also identifies when or how a student is considered “at risk” or in some way “lacking.” Second, the approach involves providing timely, individualized assistance to address these deficiencies. However, a considerable body of research outside our field has established that such deficit framing, by itself, leads to reactive and less productive strategies. In deficit-based frameworks, powerful student strengths, skills, and schemas—their assets—are not explicitly leveraged...